The CALL_EXTERNAL Function

IDL allows you to integrate programs written in other languages with your IDL code, either by calling a compiled function from an IDL program or by linking a compiled function into IDL’s internal system routine table:

The CALL_EXTERNAL function allows you to call external functions (written in C/C++ or Fortran, for example) from your IDL programs.
An alternative to CALL_EXTERNAL is to write an IDL system routine and merge it with IDL at runtime. Routines merged in this fashion are added to IDL’s internal system routine table and are available in the same manner as IDL built-in routines. This technique is discussed in Adding System Routines. To write a system routine, you will need to understand the IDL internals.

This topic covers the basics of using CALL_EXTERNAL from IDL, then discusses platform-specific options for the UNIX and Windows versions of IDL. It can be helpful to refer to the documentation for CALL_EXTERNAL when reading this material.

The CALL_EXTERNAL function loads and calls routines contained in shareable object libraries. Arguments passed to IDL are passed to this external code, and returned data from the external code is automatically presented as the result from CALL_EXTERNAL as an IDL variable. IDL and the called routine share the same process address space. Because of this, CALL_EXTERNAL avoids the overhead of process creation of the SPAWN routine. In addition, the shareable object library is only loaded the first time it is referenced, saving overhead on subsequent calls.

CALL_EXTERNAL is much easier to use than writing a system routine. Unlike a system routine, however, CALL_EXTERNAL does not check the type or number of parameters. Programming errors in the external routine are likely to result in corrupted data (either in the routine or in IDL) or to cause IDL to crash. See Common CALL_EXTERNAL Pitfalls for help in avoiding some of the more common mistakes.

Example Code in the IDL Distribution

This section contains examples of CALL_EXTERNAL use. All of the code for these examples, along with additional examples, can be found in the call_external subdirectory of the external directory of the IDL distribution. The C language examples use the MAKE_DLL procedure, and can therefore be easily run on any platform supported by IDL. To build the sharable library containing the external C code and then run all of the provided examples, execute the following IDL statements:

PUSHD, FILEPATH(’’,SUBDIRECTORY=[’external’,’call_external’,’C’]) ALL_CALLEXT_EXAMPLES

POPD

Additional information on these examples, including details on running the individual examples, can be found in the README file located in that directory.

CALL_EXTERNAL Compared to UNIX Child Process

In many situations, a UNIX IDL user has a choice of using the SPAWN procedure to start a child process that executes external code and communicates with IDL via a pipe connecting the two processes. The advantages of this approach are:

Simplicity.
The processes do not share address space, and are therefore protected from each other’s mistakes.

The advantages of CALL_EXTERNAL are:

IDL and the called routine share the same memory and data space. Although this can be a disadvantage (as noted above) there are times where sharing address space is advantageous. For example, large data can be easily and cheaply shared in this manner.
CALL_EXTERNAL avoids the overhead of process creation and parameter passing.
The shareable object library containing the called routine is only loaded the first time it is referenced, whereas a SPAWNed process must be created for each use of the external code.

Compilation and Linking of External Code

Each operating system requires different compilation and link statements for producing a shareable object suitable for usage with CALL_EXTERNAL. This is even true between different implementations of a common operating system family. For example, most UNIX systems require unique options despite their shared heritage. You must consult your system and compiler documentation to find the appropriate options for your system.

The IDL MAKE_DLL procedure provides a portable, high-level mechanism for building sharable libraries from code written in the C language. In many situations, this procedure can completely handle the task of building sharable libraries to be used with CALL_EXTERNAL. MAKE_DLL requires that you have a C compiler installed on your system that is compatible with the compiler described by the IDL !MAKE_DLL system variable.

The !MAKE_DLL system variable is used by the MAKE_DLL procedure to construct C compiler and linker commands appropriate for the target platform. If you do not use MAKE_DLL to compile and link your code, you may find the contents of !MAKE_DLL.CC and !MAKE_DLL.LD helpful in determining which options to specify to your compiler and linker, in conjunction with your system and compiler documentation. For the C language, the options in !MAKE_DLL should be very close to what you need. For other languages, the !MAKE_DLL options should be helpful in determining which options to use, as on most systems, all the language compilers accept similar options.

AUTO_GLUE

As described in Passing Parameters , CALL_EXTERNAL uses the IDL Portable Calling Convention to call external code. This convention uses an (argc, argv) style interface to allow CALL_EXTERNAL to call routines with arbitrary numbers and types of arguments. Such an interface is necessary, because IDL, like any compiled program, cannot generate arbitrary function calls at runtime.

Most C functions are not written to the IDL portable convention. Rather, they are written using the natural form of argument passing used in compiled programs. It is therefore common for IDL programmers to write so-called glue functions to match the IDL calling interface to that of the target function. On systems that have a C compiler installed that is compatible with the one described by the IDL !MAKE_DLL system variable, the AUTO_GLUE keyword to CALL_EXTERNAL can be used to instruct IDL to automatically write, compile, and load this glue code on demand, and using a cache to preserve this glue code for future invocations of functions with the same interface.

AUTO_GLUE thus allows CALL_EXTERNAL to call functions with a natural interface, without requiring the user to write or compile additional code. AUTO_GLUE is described in the documentation for “CALL_EXTERNAL” , as well as in Using Auto Glue. The examples given in Basic C Examples show CALL_EXTERNAL used with and without AUTO_GLUE.

Input and Output

Input and output actions should be performed within IDL code, using IDL’s built-in input/output facilities, or by using IDL_Message(). Performing input/output from code external to IDL, especially to the user console or tty (e.g. stdin or stdout), may generate unexpected results.

Memory Cleanup

IDL has a strict internal policy that it does not perform memory cleanup on memory it did not allocate. This is necessary so that external code which allocates memory can use any memory allocation package it desires; with this way, there is no confusion about which code is responsible for releasing allocated memory.

The code that allocates memory is always responsible for freeing it. IDL allocates and frees memory for its internal needs, and external code is not allowed to release such memory except through a proper IDL function documented for that purpose. Similarly, IDL will not intentionally free memory that it did not allocate. As such, IDL does not perform any memory cleanup calls on the values returned from external code called via the CALL_EXTERNAL routine. Because of this, any dynamic memory returned to IDL will not be returned to the system, which will result in a memory leak. Be sure to design CALL_EXTERNAL routines in a way as not to return dynamically allocated memory to IDL. Passing String Data provides an example of doing this with strings.

Memory Access

IDL and your external code share the same address space within the same running program. This means that mistakes common in compiled languages, such as a wild pointer altering memory that it does not own, can cause problems elsewhere. In particular, external code can easily corrupt IDL’s data structures and otherwise cause IDL to fail. Authors of such code must be especially careful to guard against such errors.

Argument Data Types

When using CALL_EXTERNAL to call external code, IDL passes its arguments to the called code using the data types that were passed to it. It has no way to verify independently that these types are the actual types expected by the external routine. If the data types passed are not of the types expected by the external code, the results are undefined, and can easily include memory corruption or even crashing of the IDL program.

Note: You must ensure that the arguments passed to external code are of the exact type expected by that routine. Failure to do so will result in undefined behavior.

Mapping IDL Data Types to External Language Types

When writing external code for use with CALL_EXTERNAL, your code must use data types that are compatible with the C data types used internally by IDL to represent the IDL data types. This mapping is covered in the topic IDL Internals: Types.

By-Value and By-Reference Arguments

There are two basic forms in which arguments can be passed between functions in compiled languages such as C/C++ and Fortran. To use CALL_EXTERNAL successfully, you should be comfortable with these terms and their meanings. In particular, Fortran programmers are often unaware that Fortran code passes everything by reference, and that C code defaults to passing everything by value.

By default, CALL_EXTERNAL passes arguments by reference (unless this behavior is explicitly altered by the use of the ALL_VALUE or VALUE keywords), so no special action is typically required to call Fortran code via CALL_EXTERNAL.

Note: You must ensure that the arguments passed to external code are passed using the correct method: by value, or by reference. Failure to do so will result in undefined behavior.

Arguments Passed by Value

A copy of the value of the argument is passed to the called routine. Any changes made to such a value by the called routine are local to that routine, and do not change the original value of the variable in the calling routine. C/C++ pass everything by value, but have an explicit address-of operator (&) that is used to pass addresses of variables and get by-reference behavior.

Arguments Passed by Reference

The machine address of the argument is passed to the called routine. Any changes made to such a value by the called routine are immediately visible to the caller, because both routines are actually modifying the same memory addresses. Fortran passes everything by reference, but most Fortran implementations support intrinsic operators that allow the programmer control over this (sometimes called %LOC and %VAL, or just LOC and VAL). Consult your compiler documentation for details.

Microsoft Windows Calling Conventions

All operating system/hardware combinations define an inter-routine calling convention. A calling convention defines the rules used for passing arguments between routines, and specifies such details as how arguments of different types are passed (i.e. in registers or on the system stack) and how and when such arguments are cleaned up.

A stable and efficient calling convention is critical to the stability of an operating system, and can affect most aspects of the system:

The efficiency of the entire system depends on the efficiency of the core calling convention.
Backwards compatibility, and thus the longevity of binary software written for the platform depends on the stability of the calling convention.
Calling routines from different languages within a single program depends on all the language compilers adhering to the same calling convention. Even within the same language, the ability to mix code compiled by different compilers requires those compilers to adhere to the same conventions. For example, at the time of this writing, the C++ language standard lacks an Application Binary Interface (ABI) that can be targeted by all C++ compilers. This can lead to situations in which the same compiler must be used to build all of the code within a given program.

Microsoft Windows is unique among the platforms supported by IDL in that it has two distinct calling conventions in common use, whereas other systems define a single convention. On single-convention systems, the calling convention is unimportant to application programmers; it is of concern only to hardware designers, authors of compilers, and operating systems. On a multiple convention system, application programmers need to ensure that their code is compiled to use the proper convention and that calls to that code use the same convention. The Microsoft Calling Conventions are described below.

STDCALL

STDCALL is the calling convention used by the majority of the Windows operating system API. In a STDCALL call, the calling routine places the arguments in the proper registers and/or stack locations, and the called routine is responsible for cleaning them up and unwinding the stack.

CDECL

CDECL is the calling convention used by C/C++ code by default. This default can be changed via compiler switches, declspec declarations, or #pragmas. With CDECL, the caller is responsible for both setup and cleanup of the arguments. CDECL is able to call functions with variable numbers of arguments (varargs functions) because the caller knows the actual number of arguments passed at runtime, whereas STDCALL cannot call such functions. This is because the STDARGS routine cannot know efficiently at compile time how many arguments it will be passed at runtime in these situations.

The inconvenience of having two distinct and incompatible calling conventions is usually minor, because the header files that define functions for C/C++ programs include the necessary definitions such that the compiler knows to generate the proper code to call them and the programmer is not required to be aware of the issue. However, CALL_EXTERNAL does have a problem: Unlike a C/C++ program, IDL determines how to call a function solely by the arguments passed to CALL_EXTERNAL, and not from a header file.

IDL therefore has no way to know how your external code was compiled. It uses the STDARG convention by default, and the CDECL keyword can be used to change the default. CALL_EXTERNAL therefore relies on the IDL user to tell it which convention to use. If IDL calls your code using the correct convention, it will work correctly. If it calls using the wrong convention, the results are undefined, including memory corruption and possible crashing of the IDL program.

The default calling convention for CALL_EXTERNAL is STDCALL, whereas the default convention for the Microsoft C compiler is CDECL. Hence, Windows users must usually specify the CDECL keyword when calling such code from IDL. Non- Windows versions of IDL ignore the CDECL keyword, so it is safe to always include it in cross platform code.

Here is what happens when external code is called via the wrong calling convention:

If a STDARG call is made to a CDECL function, the caller places the arguments in the proper registers/stack locations, and relies on the called routine to cleanup and unwind the stack. The called routine, however, does not do these things because it is a CDECL routine. Hence, cleanup does not happen.
If a CDECL call is made to a STDARG function, the caller places the arguments in the proper register/stack locations. The called routine cleans up on exit, and then the caller cleans up again.

Either combination is bad, and can corrupt or kill the program. Sometimes this happens, and sometimes it doesn’t, so the results can be random and mysterious to programmers who are not aware of the issue.

When the wrong calling convention is used, the process stack can become confused. A “smashed stack” visible from the C debugger following a CALL_EXTERNAL is usually indicative of when the wrong calling convention was used.

Common CALL_EXTERNAL Pitfalls

Following are a list of common errors and mistakes commonly seen when using CALL_EXTERNAL.

The number of arguments and their types, as passed to CALL_EXTERNAL, must be the exact types expected by the external routine. In particular, it is common for programmers to forget that the default IDL integer is a 16-bit value and that most C compilers define the int type as being a 32-bit value. You should be careful to use IDL LONG integers, which are 32-bit, in such cases. See Argument Data Types for additional details.
Passing data using the wrong form: Using by-value to pass an argument to a function expecting it by-reference, or the reverse. SeeBy-Value and By- Reference Arguments for additional details.
Under Microsoft Windows, using the incorrect calling convention for a given external function. See Microsoft Windows Calling Conventions for additional details.
Failure to understand that IDL uses IDL_STRING descriptors to represent strings, and not just a C style NULL terminated string. Passing a string value by reference passes the address of the IDL_STRING descriptor to the external code. See IDL Internals: String Processing for additional details.
Attempting to make IDL data structures use memory allocated by external code, rather than use the proper IDL API for creating such data structures. For instance, attempting to give an IDL_STRING descriptor a different value by using C malloc() to allocate memory for the string and then storing the address of that memory in the IDL_STRING descriptor is not supported, and can easily crash or corrupt IDL. Although IDL uses malloc()/free() internally on most platforms, this is not part of IDL’s public interface, which could be changed at any time and without notice. Even on platforms where IDL does use these functions, its use of them is not directly compatible with similar calls made by external code because IDL allocates additional memory for bookkeeping that is generally not present in memory allocations from other sources. See IDL Internals: String Processing for information on changing the value of an IDL_STRING descriptor using supported IDL interfaces. See Memory Cleanup for more on memory allocation and cleanup.
IDL is written in the C language, and when IDL starts, any necessary runtime initialization code required by C programs is automatically executed by the system before the IDL main() function is called. Hence, calling C code from IDL usually does not require additional runtime initialization. However, when calling external code written in languages other than C, you may find that your code does not run properly unless you arrange for the necessary runtime support for that language to run first. Such details are highly system specific, and you must refer to your system and compiler documentation for details. Code that is largely computational rarely encounters this issue. It is more common for code that performs Input/Output directly.
Programming errors in the external code. It is easy to make mistakes in compiled languages that have bad global consequences for unrelated code within the same program. For example, a wild memory pointer can lead to the corruption of unrelated data. If you are lucky, such an error will immediately kill your program, making it easy to locate and fix. Less fortunate is the situation in which the program dies much later in a seemingly unrelated part of the program. Finding such problems can be difficult and time consuming. When IDL crashes following a call to external code, an error in the external code or in the call to CALL_EXTERNAL is the cause in the vast majority of cases.
Some compilers and operating systems have a convention of adding leading or trailing underscore characters to the names of functions they compile. These conventions are platform specific, and as they are of interest only to system linker and compiler authors, not generally well documented. This is usually transparent to the user, but can sometimes be an issue with inter language function calls. If you find that a function you expect to call from a library is not being found by CALL_EXTERNAL, and the obvious checks do not uncover the error (usually a simple misspelling), this might be the cause. Under UNIX, the nm command can be helpful in diagnosing such problems.
C++ compilers use a technique commonly called name munging to encode the types of method arguments and return values into the name of the routine as written to their binary object files. Such names often have only a passing resemblance to the name seen by the C++ programmer in their source code. IDL can only call C++ code that has C linkage, as discussed in “C++”. C linkage code does not use name munging.
When calling external code written in other languages, there are sometimes platform and language specific hidden arguments that must be explicitly supplied. Such arguments are usually provided by the compiler when you work strictly within the target language, but become visible in inter-language calls. An example of this can be found in “Hidden Arguments”. In this example, the Fortran compiler provides an extra hidden length argument when a NULL terminated string is passed to a function.